Virtual Tool Manipulation System

ABSTRACT

In the field of virtual reality, virtual tool manipulation systems and related methods and software are described in the present disclosure. One implementation of a virtual tool manipulation system, among others, comprises a motion tracking system configured to generate motion information related to the position of a part of a user&#39;s body. The virtual tool manipulation system also comprises a haptic feedback system configured to provide a haptic sensation to the user based on the motion information, the position of a virtual tool, and characteristics of the virtual tool.

TECHNICAL FIELD

The present disclosure generally relates to virtual reality and moreparticularly relates to simulating tools that can be manipulated in avirtual environment.

BACKGROUND

During training, an apprentice learns how to use tools associated with aparticular trade. Usually, the training process involves the apprentice,under supervision, practicing with the actual tools to conduct aspecific action upon a subject. For example, the subject in this respectcan be a living organism or an inanimate object, particularly dependentupon the type of tool being used. Tool manipulation skills can bedeveloped, for instance, until the apprentice can become adequatelyproficient at the trade. Even in the medical field, for example, amedical student or surgeon-in-training learns the skills of handlingspecialized tools for performing different types of surgical orinterventional procedures.

In medical training, the surgeon-in-training typically learns the art oftool handling on a cadaver, animal, or box-type trainer. However, inrecent years, systems have been developed that allow a trainee topractice surgical procedures in a virtual environment where no realbodies are needed. In some virtual reality systems, the actual handle ofa surgical tool is removed from the rest of the tool. Sensors are thenattached to the surface of the handle to detect the position andorientation of the tool in a three-dimensional space. An interfaceallows the sensors to communicate with a computer and informationrelated to how the handle is manipulated is transferred to the computerfor further processing. Images of the tool in a virtual realm aredisplayed on a visual display device to simulate, in a visual sense, howan actual tool might affect the subject in reality.

One disadvantage of the conventional virtual reality system, in whichsensors are attached to the surface of a handle of the tool, is thatspecific simulation hardware is required for each tool. Thus, it can bevery expensive to configure the appropriate sensing hardware andsoftware for a large number of different tools. In order to overcomethese and other deficiencies of conventional systems, and to morerealistically simulate tool-handling procedures, further improvementscan still be made in the field of virtual reality involving themanipulation of tools. Not only can improved systems provide a traineewith more realistic training, but also improvements can be made toprovide additional benefits, as well as improved tool designmethodologies and rapid prototyping of new instruments.

SUMMARY

The present disclosure describes systems, methods, and associatedsoftware applications for simulating virtual tools and for virtuallyhandling the virtual tools. According to one embodiment among manydescribed herein, a virtual tool manipulation system comprises a motiontracking system configured to generate motion information related to theposition of a part of a user's body. The virtual tool manipulationsystem further comprises a haptic feedback system configured to providea haptic sensation to the user based on the motion information, theposition of a virtual tool, and characteristics of the virtual tool.

Other features, advantages, and implementations of the presentdisclosure, not expressly disclosed herein, will be apparent to one ofordinary skill in the art upon examination of the following detaileddescription and accompanying drawings. It is intended that such impliedimplementations of the present disclosure be included herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The components in the following figures are not necessarily drawn toscale. Instead, emphasis is placed upon clearly illustrating the generalprinciples of the present disclosure. Reference characters designatingcorresponding components are repeated as necessary throughout thefigures for the sake of consistency and clarity.

FIG. 1 is a block diagram of a virtual tool manipulation systemaccording to one embodiment.

FIG. 2 is an illustration showing a view of a human hand positioned togrip a virtual tool.

FIG. 3 is a block diagram of the processing system shown in FIG. 1according to one embodiment.

FIG. 4 is a block diagram of the memory shown in FIG. 3 according to oneembodiment.

FIG. 5 is a block diagram illustrating the simulation files shown inFIG. 4 according to one embodiment.

FIG. 6 is a block diagram illustrating the interaction modules shown inFIG. 4 according to one embodiment.

FIG. 7 is a block diagram illustrating the output processing modulesshown in FIG. 4 according to one embodiment.

FIG. 8 is an illustration showing a virtual image presented by thevisual feedback system shown in FIG. 1.

DETAILED DESCRIPTION

Virtual training for surgical procedures has provided a means forvisually displaying images of what a surgery might actually look like inreality. In conventional systems, the handle of an actual surgical toolis removed from the tool and a number of sensors are placed on thehandle to detect the position and orientation of the surgical tool in avirtual realm. However, the cost involved in creating a handle sensordevice and associated software for interpreting movement of the devicecan be very expensive, especially when sensor devices and associatedhardware and software is needed for each tool.

The present disclosure provides a lower cost alternative to theconventional systems by allowing any tool design, even new tool designs,to be simulated. More particularly, the design of a tool—either anexisting tool or a design of a new tool—can be entered into memory usinga computer-aided design (CAD) program. Using the design of a tool inadditional to other relevant characteristics of the tool, e.g. weight,weight distribution, center of gravity, degrees of freedom, sharpness,resiliency, etc., the present system can provide haptic or kinestheticfeedback to the user to give the user the sensation of the tool'scharacteristics, as if the user were actually holding a real tool. Thehaptic feedback gives the user the sensation of position, movement,tension, etc. of the virtual tool.

By simulating the tool in the virtual world, there is no need toactually manufacture a prototype of the tool during design development.According to the teachings of the present disclosure, the user canexperience in the virtual world how a tool functions and how it can behandled, while receiving feedback confirming the user's control of thevirtual tool. With the ability to simulate tools in this way, aplurality of different types and designs of tools can be simulated andstored. The simulated virtual tools can have different handles,different sizes, shapes, forms, textures, etc. Also, depending on thetype of tool being simulated, the tool can be virtually “gripped” in anysuitable manner according to the particular design. As a result ofeither eliminating the need to manufacture a prototype or dramaticallyreducing the number of prototypes during the R&D phase, and eliminatingthe need to create associated sensing devices and software for eachindividual tool, the cost of including numerous tools in a trainingenvironment can be greatly reduced compared to systems in which sensingstructure and related software for each physical tool must beconfigured.

Regarding new tool design according to the present disclosure, designsof tools can be entered into a computer and virtually manipulated. Withhaptic feedback provided to the user, the user is able to experience thesensation of actually holding the tool. Based on the ease of use and thefeel of the tool for accomplishing its intended purpose, the design ofthe tool can then be altered as necessary without the need tomanufacture the tool from inception. In this sense, since no physicalmaterial is needed, example embodiments described herein can avoidwasting manufacturing and material costs. The iterative process ofdesigning a tool, utilizing the tool in the virtual realm, and tweakingthe design of the tool, can result in a better tool design and can beaccomplished in a shorter amount of time.

FIG. 1 is a block diagram of an embodiment of a virtual toolmanipulation system 10. In this embodiment, virtual tool manipulationsystem 10 includes a motion tracking system 12, a processing system 14,a haptic feedback system 16, and a visual feedback system 18. In someembodiments, virtual tool manipulation system 10 may also include anaudio feedback system (not shown) to provide audio to the user, ifdesired. For instance, audio feedback may be beneficial to enhance thevirtual experience, especially for tools having parts that producecertain sounds, e.g. motors, joint movement, friction, vibration,activation of an instrument, etc.

Virtual tool manipulation system 10, according to the teachings herein,can be configured for simulating any type of operation involving humanmovement, e.g. computer gaming, surgery, and involving any occupation,e.g. surgeon, aircraft pilot, astronaut, scientist, construction worker,factory worker, etc. Virtual tool manipulation system 10 can be used fortraining purposes to allow a specialist to practice a procedure in asafe training setting.

Motion tracking system 12 may include sensing devices for tracking thekinematics or position of certain points in three-dimensional space overtime. The sensing device can also track the position or angle of thesepoints with respect to each other, or using other motion trackingtechniques. In particular, motion tracking system 12 is capable ofmaking several measurements of position every second to simulatecontinual movement. In some embodiments, motion tracking system 12 doesnot include a corresponding physical element that the user can hold ortouch in reality. Instead, in this case, the virtual tool exists only inthe virtual realm and is completely virtual. In other embodiments,motion tracking system 12 includes a physical prop that the user cantouch. The physical prop in this case can be, for example, a handlehaving the approximate size and shape as the virtual tool.

Motion tracking system 12, for example, may include a data glove, suchas a CyberGlove™, which can be worn on the hand of a user. The dataglove can include sensors for measuring the bending angles of severaljoints of the user's hand, wrist, and fingers. In some implementations,motion tracking system 12 may be capable of measuring x, y, and zpositions in three-dimensional space with respect to a reference pointand may further be capable of measuring orientation in space relative toa fixed reference frame. In other embodiments, motion tracking system 12may include an optical motion capture device for tracking images of theuser and calculating the positions and orientations of various user bodyparts, such as the hand or hands of a user, wrists, arms, shoulders,head, legs, feet, etc., if necessary.

Processing system 14 may include any suitable processing and storagecomponents for managing the motion information measured by motiontracking system 12. FIG. 3 depicts processing system 14 according to oneembodiment, as described in more detail below. Using specific softwareapplications, processing system 14 can analyze the motion information todetermine how the user's motion corresponds to a human manipulator inthe virtual world. The movement of the human manipulator with respect tovirtual tools stored in processing system 14 can be determined. As thehuman manipulator manipulates the virtual tool, the processing system 14can also determine the result of an interaction between the virtual tooland a virtual subject. Based on calculations of the interaction betweenthe manipulator and the virtual tool, processing system 14 calculateshaptic feedback signals to be applied to haptic feedback system 16.Also, processing system 14 calculates video signals that are applied tovisual feedback system 18 to display to the user the virtual image ofthe interactions among the virtual manipulator, virtual tool, andvirtual subject.

Haptic feedback system 16 may include any suitable device that providesany type of forced feedback, vibrotactile feedback, and/or tactilefeedback to the user. This feedback is able to provide the user with thesensation of actually holding the simulated tool. Haptic feedback system16 simulates the physical texture, pressures, forces, resistance,vibration, etc. of the tool, which can be related in some respects toresponses to the tool's movement in space and including the interactionof the tool with the subject. Haptic feedback system 16 may includemechanisms such as the CyberGrasp™, CyberForce™, CyberTouch™, etc., forexample, which can apply at least the sensations of force, weight,resistance, and vibration to the user.

Visual feedback system 18 may include any suitable virtual realitydisplay device, such as virtual goggles, display screens, etc. Visualfeedback system 18 can show the appearance of the tool and how the toolperforms when handled by the virtual manipulator. Visual feedback system18 may also show how the tool reacts to various forces or actionsapplied to the tool, and how the subject is affected by the virtualtool's movements or operations.

Generally, virtual tool manipulation system 10 operates as follows.Motion tracking system 12 tracks the motion of one or more parts, e.g.hands, of a user's body. The motion information is sent to processingsystem 14, which processes the motion information and determines how themotion affects a virtual tool. Also, processing system 14 determines howthe virtual tool interacts with a particular subject. In response tothese processing procedures, processing system 14 provides hapticfeedback signals to haptic feedback system 16 based on interactionsbetween the user and the virtual tool depending on the particular motionof the user and characteristics of the virtual tool. Also, processingsystem 14 provides virtual reality information to visual feedback system18 to provide real-time visual simulation of the user's hand, tool beingmanipulated, and the effect of the tool's movement on the subject. Inaddition, processing system 14 may also create audio signals related tothe interactions among the virtual manipulator, virtual tool, andvirtual subject.

FIG. 2 illustrates a view of a real human hand of a user 20 who is usingvirtual tool manipulation system 10 of FIG. 1. In this view, user 20 hashis or her hand positioned in a “Palmar” grip or “dinner knife” grip,which may be used for handling certain tools, e.g. scalpels, lancets,etc. In other embodiments, the hand of user 20 could be positioned inanother grip, e.g. a pencil grip, depending on the type of virtual toolbeing handled. User 20 can represent any human practicing or training inthe use of tools or other manipulatable objects. User 20 can be, forexample, a surgeon, medical student, pilot in training, scientist,astronaut in training, construction worker, machine operator, factoryworker, etc. In the medical field, for example, the tools that may bevirtually gripped by user 20 may include forceps, clamps, scalpels,lancets, trocars, retractors, dermatomes, endoscopes, laparoscopictools, etc. Depending on the particular field, user 20 may virtuallymanipulate other types of tools.

In some embodiments, user 20 does not actually grasp an actual tool oreven a handle of an actual tool, which is unlike conventional systemsthat normally require the use of a tool handle altered to includesensors thereon. In other embodiments, user 20 may grasp a physical propthat resembles the size and/or shape of an actual tool handle. Althoughnot specifically shown in this figure, motion tracking system 12(FIG. 1) tracks the motion of user 20 to determine the positioning ofthe hand, fingers, thumbs, wrist, etc. In some embodiments, motiontracking system 12 may include a data glove, such as a CyberGlove™ orCyberForce™ for sensing positional and/or rotational characteristics ofthe hand of user 20. In many embodiments, tool manipulation involves ahuman's hand, but, in other implementations, both hands may be used oreven other parts of the human, such as the wrists, arms, head, legs,feet, etc. For example, the positioning of the head of the human can betracked to determine the angle and line of sight of the human's eyesupon the virtual scene, thereby affecting how the visual signals aredisplayed by visual feedback system 18.

Also, although not specifically shown, user 20 and/or a data glove wornon the hand of user 20 can receive haptic feedback from haptic feedbacksystem 16 (FIG. 1). For example, haptic feedback system 16 may includeany suitable force feedback mechanisms, such as the CyberGrasp™,CyberForce™, CyberTouch™, etc., which can operate in conjunction with adata glove. Particularly, haptic feedback system 16 may include variousactuators for simulating the feel of the virtual tool as if it were anactual tool. Also, based on other characteristics of the virtual tool,such as weight, resilience, degrees of freedom, etc., haptic feedbacksystem 16 can provide the sensation of manipulating the virtual tool andprovide sensory feedback based on how the tool interacts with thesubject. Therefore, haptic feedback system 16, depending on the type oftool is being simulated, can provide the sensation of pressure, forces,tendency to resist movement, gravity, vibration, inertia, etc.

FIG. 3 is a block diagram of an embodiment of processing system 14 shownin FIG. 1. In this embodiment, processing system 14 includes amicroprocessor 22, memory 24, input/output devices 26, motion trackerinterface 28, haptic device interface 30, virtual display interface 32,each interconnected by an internal bus 34 or other suitablecommunication mechanism for communicating information. Processing system14 may also include other components and/or circuitry associated withprocessing and computing digital or analog electrical signals.Processing system 14 may be configured as a computer that is capable ofprocessing motion control signals detected from a user and providingfeedback to the user to simulate the experience of a virtual encounter.

Microprocessor 22 may be a general-purpose or specific-purpose processoror microcontroller. Memory 24 may include internally fixed storageand/or removable storage media for storing information, data, and/orinstructions. The storage within the memory components may include anycombination of volatile memory, such as random access memory (RAM),and/or non-volatile memory, such as read only memory (ROM). Memory 24can also store a software program enabling microprocessor 22 to executea virtual tool manipulation program or procedure. Various logicalinstructions or commands may be included in the software program foranalyzing the user's movements and regulating feedback to the user basedon the virtual interactions among the virtual human manipulator, thevirtual tool, and the virtual subject. The virtual tool manipulationprogram of the present disclosure can be implemented in hardware,software, firmware, or a combination thereof. When implemented insoftware or firmware, the virtual tool manipulation program can bestored in memory 24 and executed by microprocessor 22. When implementedin hardware, the virtual tool manipulation program can be implemented,for example, using discrete logic circuitry, an application specificintegrated circuit (ASIC), a programmable gate array (PGA), a fieldprogrammable gate array (FPGA), etc., or any combination thereof.

In the embodiment of FIG. 3, memory 24 is illustrated as being part ofprocessing system 14. In other embodiments, parts or all of the memorycomponents associated with processing system 14 may be configured inother processing systems or incorporated on a removable storage device.Also, memory 24 may also include remote storage that is accessible via amodem or other network communication device.

Memory 24 includes files that include information for simulating variousportions of the virtual tool environment. For example, the simulationfiles may include a simulation of the tool itself, a simulation of themanipulator of the virtual tool, and a simulation of the subject that isimpacted by the tool's operation. Memory 24 can also include softwareprograms or code for defining how the virtual tool interacts with otherthings. For example, the interaction between the manipulator and tool aswell as the interaction between the tool and the subject can be defined.

Input/output (I/O) devices 26 include input mechanisms such askeyboards, keypads, cursor control devices, e.g. computer mice, or otherdata entry devices. Output devices may include a computer monitor,display device, printer, or other peripheral devices. The I/O devices 26may also include a device for communicating with a network, such as amodem, for allowing access to the network, such as the Internet. The I/Odevices 26 can communicate with internal bus 34 via wired or wirelesstransmission.

Motion tracker interface 28 receives information received by motiontracking system 12 (FIG. 1). This information may be stored temporarilyor long term in memory 24 and processed to determine the position and/ororientation of user 20. Microprocessor 22 executes the virtual toolmanipulation program by analyzing the motion detection information ofuser 20 and determining the effect of this movement on a virtual tooland the effect of the tool on the virtual subject. Based on thesemovements and interactions, microprocessor 22 determines feedbacksignals intended to be applied to user 20. Haptic device interface 30transfers haptic feedback signals to haptic feedback system 16 tosimulate for the user 20 the tactile sensation of handling andcontrolling the virtual tool. Virtual display interface 32 transfersvideo signals to visual feedback system 18 to simulate for the user 20the virtual reality images of the virtual manipulator using the virtualtool on the virtual subject.

FIG. 4 is a block diagram of an embodiment of memory 24 shown in FIG. 3.In this embodiment, memory 24 includes, among other things, a virtualtool manipulation program 38, an operating system (O/S) 40, and acomputer-aided design (CAD) program 42. The virtual tool manipulationprogram 38 includes, among other things, simulation files 44,interaction modules 46, and output processing modules 48. In someembodiments, memory 24 may include other software applications, such asan Internet browser, word processing program, etc. The CAD program 42can be installed in memory 24 to enable a tool designer to enter thefeatures of a tool design, which can be stored as the simulation files.Information pertaining to virtual tool manipulator simulation andvirtual subject can also be entered using CAD program 42 or other inputmechanism. In addition to the tool design, simulation files 44 may alsoinclude physical and operational parameters of the tools. In otherembodiments, CAD program 42 may be omitted and existing tool designfiles can be loaded into memory 24 from an external source. Also, tooldesign files can be downloaded from the Internet and stored in memory24.

The virtual tool manipulation program 38 manages the simulation of thevirtual tool, the simulation of the tool's manipulator, and the subjectof the tool's operation, each stored as simulation files 44. The virtualtool manipulation program 38 also manages how the tool interacts withthe manipulator and subject, depending on characteristics and propertiesof the manipulator, tool, and subject defined in simulation files 44.The manner in which they interact can be modeled and defined usingsoftware code and stored as interaction modules 46. Output processingmodules 48 of virtual tool manipulation program 38 includes softwarecode relating to how information is fed back to the user 20. Based onthe virtual interactions of the manipulator, tool, and subject frominteraction modules 46, output processing modules 48 provide at leasthaptic and visual feedback for simulating the virtual tool manipulationexperience. Virtual tool manipulation program 38 and any other programsor software code including executable logical instructions as describedherein can be embodied in any suitable computer-readable medium forexecution by any suitable processing device. The computer-readablemedium can include any physical medium that can store the programs orsoftware code for a measurable length of time.

FIG. 5 is a block diagram of an embodiment of the simulation files 44shown in FIG. 4. In this embodiment, the simulation files 44 include amanipulator simulation file 50, a tool simulation file 52, and a subjectsimulation file 54. Manipulator simulation file 50 includes informationdefining a virtual human operator. Specifically, since most tools aremanipulated by a user's hands, the hands, wrists, and forearms can besimulated with respect to shape, movement, rotation, bending,kinematics, etc. Manipulator simulation file 50 includes various rangesof motion of joints and bones that exist in a human's hand, fingers,wrists, arms, etc. Also included are relative movement information withrespect to different joints of a finger and relative movement of onefinger with respect to another. Manipulator simulation file 50 can modelthe human hand and wrist to allow six or more degrees of freedomtracking information to position the virtual hand(s) at any point in thevirtual space and oriented at any reasonable angle.

In other embodiments, manipulator simulation file 50 may include otherinformation defining other parts of the human body. For example, if atool to be simulated involves the manipulator pressing a pedal with hisor her foot, then manipulator simulation file 50 includes informationregarding various foot positions and orientations. In this respect,manipulator simulation file 50 may include data defining two or moreparts of the human body. For some tools, two hands may be needed toutilize the tool properly. In this case, two files, one for each hand,can be modeled. Other implementations of manipulator simulation file 50can account for variations in the size of the user. For example, themanipulation simulation file 50 may allow modifying the simulationinformation for a manipulator with large hands, for example.

Tool simulation file 52 stores information or data regarding theparameters, characteristics, and functions of the particular tool beingsimulated. The information in tool simulation file 52 can be downloadedfrom an external source or can be produced locally using the CAD program40. The information includes simulation data of all parts of the virtualtool, including the manipulated parts, e.g. handles, buttons, switches,pulls, etc., and other parts of the structure of the tool, e.g. blades,clamping jaws, tool head, etc. Tool simulation file 52 also includesinformation regarding the characteristics of the tool, such as size,shape, form, sharpness, texture, force impact, vibration, inertia,center of gravity, weight, weight distribution, degrees of freedom,angles of motion, etc. Other characteristics such as resistance tomovement, flexibility, inertia, etc. can also be used to define thetools.

Depending on the type of tool being simulated, tool simulation file 52may differ significantly from one type to another. For example, virtualsurgical tools could include scalpels, lancets, clamps, forceps,retractors, hemostats, dermatomes, endoscopes, laparoscopy tools, etc.Carpentry and construction tool simulated in the tool simulation filemay include, for example, hammers, saws, drills, picks, crowbars, etc.Machine operation controls, such as machines for controllingconstruction equipment, vehicles, factory equipment, etc. can also besimulated and stored as a tool simulation file 52. It should beunderstood that tool simulation file 52 can include any amount ofinformation for a number of simulated tools. For instance, for surgicaltraining, a surgeon may need certain types of tools for certainprocedures. These tools can be stored in related tool simulation filesfor access as needed. The characteristics of existing tools can beloaded in tool simulation file 52, or, as suggested above, new tools canbe designed using appropriate software, such as CAD program 42 (FIG. 4).The physical design as well as the functional design of each of thetools can be stored in tool simulation file 52.

Subject simulation file 54 includes information or data defining thevirtual subject that receives the actions of the virtual tool. Thesubject can be animate or inanimate, depending on the types of toolsbeing simulated. For surgical tools, the subject would normally beanimate, such as a human or, in the case of veterinary medicine, ananimal. The anatomy and physiology of the human or animal can be modeledand stored in subject simulation file 54. The anatomy may include thebody, skin, tissues, organs, bones, muscles, ligaments, tendons, etc.Other parts of the anatomy may also be incorporated in subjectsimulation file 54 as needed, depending on the type of virtual procedurethe user is performing. Subject simulation file 54 may includecharacteristics of the subject, such as flexibility, strength,resistance to movement, inertia, resilience, ability to be cut orpunctured, etc. Also, subject simulation file 54 can be implementedhaving a range of characteristics as needed. For example, differences insize, shape, age, sex, etc. of a patient can be modeled to better matchthe type of surgical procedure with an appropriate body type.

In some embodiments, a model can be adapted or derived from CAT scans,CT scans, x-rays, etc. of a specific patient to simulate a virtualrepresentation of the patient to be operated on. In this sense, asurgeon could practice a surgical procedure on the virtual patient tobetter understand the real conditions of the patient and any risks orcomplications that may be involved. Also, subject simulation file 54 caninclude the differences in anatomy between a male patient and femalepatient if applicable to the particular surgical procedure.

In the fields of building construction or machine operation, the toolswould normally be used on inanimate objects, such as lumber, drywall,beams, girders, nails, screws, rivets, or other building materials.Other subjects may include earth, dirt, rocks, concrete, etc. Thecharacteristics of the subjects can be modeled to define the subject'ssize, shape, weight, flexibility, etc. In other fields in which virtualtools can be simulated and manipulated in a virtual world, otherspecific subjects receiving the action of the tools can be simulated toreflect a similar subject in reality.

FIG. 6 is a block diagram of an embodiment of interaction modules 46shown in FIG. 4. In this embodiment, interaction modules 46 include amanipulator/tool interaction module 60 and a tool/subject interactionmodule 62. In some embodiments, the interaction modules 44 may alsoinclude a manipulator/subject interaction module 64, if the conditionmay apply in which the manipulator actually contacts the subject.Manipulator/subject interaction module 64 may help to simulate a realenvironment in which the manipulator can directly sense the subject toperceive an understanding of the position of the subject with respect toanother hand that may be holding the tool.

Manipulator/tool interaction module 60 includes software programs orcode for defining how the manipulator and tool interact with each other.Manipulator/tool interaction module 60 retrieves information from themanipulator simulator file 50 and tool simulation file 52 to determinethe characteristics of each. Based on the movements of the manipulatorand the characteristics of the tool, manipulator/tool interaction module60 determines how the tool will move in the virtual realm and what typesof haptic feedback the tool might provide to the manipulator. If a toolis grasped in an awkward manner, the haptic feedback can be helpful tothe user in that it can be sensed whether or not the grip needs to becorrected. Also, the user might be able to better understand the feel ofa tool from the haptic feedback. Also, depending on the flexibility ofthe manipulator with respect to the tool, certain forces or pressuresmay be determined to simulate the actual sensation of handling the tool.The movement and orientation of the tool depend on the forces applied bythe manipulator and may also react to other forces such as gravity andinertia.

Tool/subject interaction module 62 includes software programs and/orcode for defining how the tool and the subject interact with each other.Tool/subject interaction module 62 retrieves information from toolsimulation file 52 and subject simulation file 54 to determine thecharacteristics of each. The interaction of the tool and subject may bebased, for example, on the position, orientation, and movement of thetool. Also, based on the strength and other characteristics of thesubject, the tool may experience some resistance in its movement. Insome situations, this resistance can be applied to manipulator/toolinteraction module 60 to translate this resistance to the manipulator.If this is the case, the manipulator may be required to apply additionalforce or change his or her movement as needed. The reaction of thesubject to certain tool functions is also determined by tool/subjectinteraction module 62. For instance, when the subject is a virtualsurgery patient, the use of a tool may result in bleeding or otherreactions dependent upon the type of procedure performed on the subjectand the portion of the subject encountering the effects of the tool. Ina virtual construction setting, the subject may be virtual wood, forexample, and the virtual tool may create virtual sawdust as a result ofthe tool interaction with the wood. These and other interactions betweenthe tool and the subject can be programmed into tool/subject interactionmodule 62 to simulate a realistic reaction of the subject to the tooland any feedback that the subject may impose on the tool.

FIG. 7 is a block diagram of an embodiment of the output processingmodules 46 shown in FIG. 4. In this embodiment, the output processingmodules 46 include a haptic feedback processing module 70 and anaudio/video processing module 72. For the embodiments in which virtualtool manipulation system 10 includes the optional audio feedback systemfor providing an audio feature, the output processing modules 46 mayinclude an audio processing module. In this respect, the audioprocessing module can generate sound effects based on the variousinteractions among the manipulator, tool, and subject.

As a result of determining the interactions among the manipulator, tool,and subject, haptic feedback processing module 70 can provide feedbackof certain forces, pressures, vibrations, etc. to the manipulator. Notonly does haptic feedback processing module 70 derive haptic signalsfrom manipulator/tool interaction module 60 and manipulator/subjectinteraction module 64, but also haptic feedback processing module 70 canderive indirect haptic signals from tool/subject interaction module 62that may result from the interaction between the tool and the subject.Haptic signal processed in haptic feedback processing module 70 can betransmitted to haptic feedback system 16 (FIG. 1) via haptic deviceinterface 30 (FIG. 3). Based on the type of haptic feedback system 16 inuse, haptic feedback processing module 70 alters the haptic signals toenable haptic feedback system 16 to provide appropriate feedback to theuser.

The output processing modules 46 also include the video processingmodule 72, which generates the virtual reality video image signals. Thevideo processing module 72 may include a graphics processing device forrendering the objects in the three-dimensional virtual world to atwo-dimensional display screen. The video image signals generated by thevideo processing module 72 are transmitted to visual feedback system 18(FIG. 1) via virtual display interface 32 (FIG. 3).

FIG. 8 illustrates a snapshot of a real-time virtual view 80 inaccordance with one example. This virtual view 80 includes images of avirtual manipulator 82, a virtual tool 84, and a virtual subject 86. Inparticular, the virtual manipulator 82 is the human hand of a virtualsurgeon. Also, the virtual manipulator 82 is shown handling the virtualtool 84 in such a way to impose the virtual tool 84 on the virtualsubject 86. In this example, the virtual subject 86 is the anatomy of avirtual patient. Because of the tool's interaction with the virtualsubject 86, the effect of the virtual tool 84 on the virtual patient canbe illustrated as well. In this case, the virtual tool 84 is a scalpelused to cut through the skin of the virtual patient.

It should be understood that the steps, processes, or operationsdescribed herein may represent any module or code sequence that can beimplemented in software or firmware. In this regard, these modules andcode sequences can include commands or instructions for executingspecific logical steps, processes, or operations within physicalcomponents. It should further be understood that one or more of thesteps, processes, and/or operations described herein may be executedsubstantially simultaneously or in a different order than explicitlydescribed, as would be understood by one of ordinary skill in the art.

The embodiments described herein merely represent examples ofimplementations and are not intended to necessarily limit the presentdisclosure to any specific embodiments. Instead, various modificationscan be made to these embodiments as would be understood by one ofordinary skill in the art. Any such modifications are intended to beincluded within the spirit and scope of the present disclosure andprotected by the following claims.

1. A virtual tool manipulation system comprising: a motion trackingsystem configured to generate motion information related to the positionof a part of a user's body; and a haptic feedback system configured toprovide a haptic sensation to the user based on the motion information,the position of a virtual tool, and characteristics of the virtual tool.2. The virtual tool manipulation system of claim 1, further comprising aprocessing system configured to process the motion information togenerate haptic feedback signals, the processing system transmitting thehaptic feedback signals to the haptic feedback system, wherein thehaptic feedback signals are related to the haptic sensation.
 3. Thevirtual tool manipulation system of claim 2, wherein the processingsystem comprises a microprocessor, memory, a first interface device, anda second interface device, the first interface device configured tocommunicate with the motion tracking system and the second interfacedevice configured to communicate with the haptic feedback system.
 4. Thevirtual tool manipulation system of claim 1, further comprising a visualfeedback system configured to provide video images to the user based onthe motion information, the position of the virtual tool, the design ofthe virtual tool, and a virtual subject that receives the effect of thevirtual tool.
 5. The virtual tool manipulation system of claim 1,wherein the motion tracking system comprises a data glove.
 6. Thevirtual tool manipulation system of claim 1, wherein the motion trackingsystem comprises an optical motion capture system.
 7. The virtual toolmanipulation system of claim 1, wherein the motion tracking system isconfigured to track the motion of a human in training and the hapticfeedback system is configured to provide haptic feedback to the human intraining.
 8. The virtual tool manipulation system of claim 7, whereinthe human in training is a surgeon-in-training, the virtual tool is avirtual surgical tool, and a subject experiencing the actions of thevirtual surgical tool is a virtual patient.
 9. The virtual toolmanipulation system of claim 1, wherein the virtual tool does notinclude a corresponding physical structure in reality.
 10. The virtualtool manipulation system of claim 1, further comprising a physical propcorresponding to the virtual tool.
 11. A processing device comprising: amemory device adapted to store a virtual tool manipulation program; amicroprocessor adapted to execute the virtual tool manipulation program;a motion tracker interface adapted to receive motion informationrepresenting motion of a user; and a haptic device interface adapted togenerate haptic feedback signals for controlling a haptic feedbackdevice; wherein the virtual tool manipulation program is adapted tosimulate a surgical procedure.
 12. The processing device of claim 11,further comprising input/output (I/O) devices allowing a tool designerto enter a tool design into the memory device.
 13. The processing deviceof claim 11, further comprising a virtual display interface adapted togenerate video signals to a visual feedback system.
 14. The processingdevice of claim 11, wherein the virtual tool manipulation program isimplemented on a computer-readable medium.
 15. The processing device ofclaim 14, wherein: the virtual tool manipulation program comprisessimulation files, interaction modules, and output processing modules;the simulation files simulate a virtual manipulator, a virtual tool, anda virtual subject; the interaction modules simulate a first interactionbetween the virtual manipulator and the virtual tool and a secondinteraction between the virtual tool and the virtual subject; and theoutput processing modules generate signals to stimulate at least onesense of the user.
 16. The processing device of claim 15, wherein theoutput processing modules generate haptic feedback signals and virtualreality image signals.
 17. A virtual tool manipulation program stored ona computer-readable medium, the virtual tool manipulation programcomprising: simulation data configured to define characteristics of avirtual tool, a virtual tool manipulator, and a virtual subject; logicconfigured to define the interaction among the virtual tool and at leastone of the virtual tool manipulator and the virtual subject; and logicconfigured to generate output information for controlling a hapticfeedback system to induce haptic sensations on a user.
 18. The virtualtool manipulation program of claim 17, wherein: the simulation datadefining the virtual tool is configured to define a virtual surgicaltool; the simulation data defining the virtual tool manipulator isconfigured to define a virtual surgeon; and the simulation data definingthe virtual subject is configured to define a virtual patient.
 19. Thevirtual tool manipulation program of claim 17, wherein the interactionmodules comprise a manipulator/tool interaction module for analyzing theinteraction between the virtual tool manipulator and the virtual tooland a tool/subject interaction module for analyzing the interactionbetween the virtual tool and the virtual subject.
 20. The virtual toolmanipulation program of claim 19, wherein the interaction modulesfurther comprise a manipulator/subject interaction module for analyzingthe interaction between the virtual tool manipulator and the virtualsubject.
 21. The virtual tool manipulation program of claim 17, wherein:the logic configured to generate output information comprises a hapticfeedback processing module and a virtual reality processing module; thehaptic feedback processing module is configured to generate hapticfeedback signals for controlling the haptic feedback system; and thevirtual reality processing module is configured to generate videosignals to a visual feedback system that displays images to the user.22. The virtual tool manipulation program of claim 17, wherein theoutput processing modules further comprise an audio processing moduleconfigured to generate audio signals for providing audio output to theuser.
 23. The virtual tool manipulation program of claim 17, wherein thevirtual tool exists exclusively in a virtual realm.